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ABSTRACT 

Problems associated with low response rates to 
surveys are considered, drawing from the literature on the 
methodology of survey research. A series of analyses are presented 
which were designed to examine the efficacy of Astin and Molm's 
procedure to adjust for nonresponse biases. Data were obtained form 
the Cooperative Institutional Research Program survey of 1987 
incoming students and the 1991 followup survey. Data were analyzed 
for 209,627 students at 390 institutions. After separating the data 
into respondents (21 percent) and nonrespondent s (79 percent) , 
attention was directed to how distributions and relationships change 
due to nonresponse. The efficacy of the Astin and Molm procedure was 
examined using univariate and joint distributions and multiple 
regressions. It is concluded that the Astin and Molm procedure for 
adjusting for patterns of nonresponse is very effective in reducing 
biases in univariate distributions. The effectiveness of this 
weighting procedure is less clear in adjusting the results of 
correlation and regression analyses. The analyses did not reveal a 
situation where the weighted results were clearly less preferable 
than the unweighted one. Although the Astin and Molm technique is 
designed for use in longitudinal research, related methods of 
weighting can be implemented for cress-sect ional surveys . (Contains 
27 references.) (SW) 
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Working with low survey response rates: 
The efficacy of weighting adjustments 



Eric L. Dey 

Center for the Study of Higher and Postsecondary Education 
University of Michigan 

Among survey researchers, obtaining a high response rate is akin to reaching nirvana. 
Although the figure that defines a high response rate is somewhat dependent upon the eye of the 
beholder, one trend is not: Americans are increasingly less likely to be willing to respond to 
surveys^ (Groves, 1989; Steeh, 1981). This trend is quite troubling since surveys play a central 
role in the data collection activities of most institutional research efforts (Grosset, 1995; Schiltz, 
1988; Cote, Grinnell, & Tompkins, 1986). 

Within the context of higher education research, an example of decluiing response rates is 
provided by a series of national longitudinal follow-up surveys of college students conducted by 
the American Council on Education and the Cooperative Institutional Research Program (CIRP). 
As can be clearly seen in Table 1, there has been a pronounced decline in response rates since the 
early 1960s, and response rates appear to be continuing their decline at the rate of a few percentage 
points a year Although the national CIRP surveys provide a good way of documenting this decline 
due to their continuity, the response rate issue affects many single-institution research projects as 
well (see for example Cabrera, Castaneda, Nora, & Hengstler, 1992, who report a response rate of 
19 percent). 

Although telephone and face-to-face surveys are commonly used in large-scale public 
opinion research (Groves, 1989; Davis & Smith, 1991), mail surveys are more commonly used 
within higher education settings (Grosset, 1995; Dillman, 1991; Schiltz, 1988; Cote, Grinnell, & 

* Based on data from the General Social Survey, Davis & Smith (1992) disagree with this point but acknowledge thai 
declining rates of survey cooperation may be masked by improved field procedures in the GSS. 
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Tompkins, 1986) and will serve as the focus of this paper. This is an important distinction when 
considering the issue of nonresponse, since mail surveys have, on average, a much lower rate of 
response than that associated with other modes of data collection. Although this has long been 
recognized, the mail survey has continued to be popular in many settings due to low costs and ease 
of administration (Grosset, 1995; Dillman, 1991). 

Although there is a wealthy literature available to institutional researchers that suggest ways 
to dramatically improve response rates in mail survey (e.g., Dillman, 1978, 1983, 1991; Cote, 
Grinnell, & Tompkins, 1986; Smith & Bers, 1987), we also need to have methods of dealing with 
data generated by survey efforts when the rates of response are below — and sometimes well below 
- 100 percent. One long-standing procedure for accomplishing this is the use of weighting 
techniques to compensate for errors in survey coverage and unit (as opposed to item) nonresponse, 
with the weight representing the inverse of the probability of being sampled and responding, 
respectively (Kalton, 1983; Oh & Scheuren, 1983; Rossi, Wright, & Anderson, 1983). 

Within the field of higher education, Astin and Molm (1972) have described a method of 
dealing with survey nonresponse in cases where researchers have information on the characteristics 
of individual respondents (as is the case with longitudinal research efforts). This method, which 
has been used for many years in the CIRP follow-up surveys, employs multiple regression to 
calculate the likelihood that a student will return a completed survey. Using this information, it is 
possible to produce a weighting factor that consists of the inverse of a student's computed 
probability of response. This weighting factor thus adjusts the analyses by giving "the greatest 
weight to the responses of those students who most resemble the non-respondents" (Higher 
Education Research Institute, 1992). 

Although the Astin and Molm procedure has been employed for many years, it should be 
noted that it was developed at a time when response rates to follow-up surveys employing similar 
administration modes were two to three times what they currentiy are (see Table 1). Thus, it may 
be that the Astin and Mohn (and similar) weighting adjustments may no longer be as effective at 
compensating for nonresponse. Set within this context^ then^ this paper has several main goals. At 
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the outset, I review the analytical challenges associated with low response rates by drawing upon 
the general literature on the niethodology of survey research, I then present a series of analyses 
designed to examine the efficacy of the Astin and Molm procedure as a general method for 
adjusting for nonresponse biases, and conclude with a set of recommendations to consider in order 
to improve the quality of survey research within the context of institutional research. 

Problems associated with survey nonresponse 

In understanding the problems associated with nonresponse to mail surveys, it is important 
to recognize that response rate is different from response error or bias. It may be that a survey 
which yields a very low response rate, say 10 percent, does a fairly good job of representing the 
population from which the mailout sample was originally drawn. This would be the case if the 10 
percent who responded to this hypothetical survey are quite similar to the 90 percent who failed to 
respond. It is not often the case, however, that respondents are a perfectly random subset of those 
to whom surveys were originally mailed (although in many situations this is an untestable 
proposition), which leads to the problem of nonresponse bias (Dillman, 1991), 

The effect of nonresponse bias is manifested in different ways for different statistics, but 
can be fairly easily visualized by examining the nonresponse bias associated with a sample mean. 
For a particular variable (Y), the sample mean is calculated as follows: 

Y = PrYr + PnYn 

where pr and pn represent the proportions of respondents and nonrespondents, respectively 
(following Davis & Smith, 1991). Put another way, the nonresponse bias is "a function of the 
percentage of the sample not responding to the survey and the difference on the statistic between 
respondents and nonrespondents** (Groves, 1989, p, 134), or 
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nonresponse bias = response rate * (Yt - ¥„) 



From these equations it is clear that the extent to which Yr and Yn are different from one another 
will be directiy related to how well the n>ean among respondents (Yr) represents the true mean in 
the sample (Y). More complex statistics are affected by nonresponse bias as well, but in a less 
intuitively obvious way. In the case of a simple regression under assumptions of normality, the 
relationship between the standardized regression coefficient in the original sample (P) and that 
which is estimated from survey respondents (P ) is: 

, p v\y)/v(y) 

where V*iy) is the variance of the dependent variable in the original sample, V(y) is the variance of 
the dependent variable aixx>ng survey respondents, and is the coefficient of determination for the 
regression nxxiel among survey respondents (from Goldberger, 1981, cited in Groves, 1989). It is 
important to note that while response rate plays a straightforward role in the case of sample means, 
the nonresponse bias is simply the function of variances in the original and respondent samples and 
of model fit. 

In most cross-sectional surveys it is impossible to precisely know the degree to which 
statistics may be biased by nonresponse due to the simple fact that littie is known about the 
characteristics of nonrespondents. Although it is common in such surveys to compare respondents 
and nonrespondents across known population characteristics (such as basic demographic 
information, or within the context of research on students grade point average), the reality is that 
**even when one can be confident that no differences exist on these [types of] variables, one still 
does not know whether differences exist on those variables of interest that led to the decision to 
conduct tiie survey" (Dillman, 1991, p. 299). This has led to tiie response rate being seen as a 
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proxy for nonrespon^ bias despite the lack of correspondence between these two concepts 
(DiUman, 1991; Groves, 1989)» 

The Astin and Molm (1972) procedure exploits the fact that additional information is 
available on those to whom follow-up surveys were sent, regardless of their later response status. 
In the context of the CIRP follow-up surveys, the nonresponse weights are generated by 
regressing a variable denoting the final survey response status (1 = respondent, 0 = 
nonrespondent) upon a large number of variables derived from the CIRP freshman survey and an 
independent survey of coUege registrars. By employing regression it is possible to generate 
weights based on a large number of student characteristics (many of which may be related to one 
another) (see HERI, 1992)» Although this should in theory produce more precise weights for 
individuals, it is conceptually similar to other weighting procedures which usually rely on a limited 
number of variables to create stratification cells (Rossi, Wright, & Anderson, 1983). Given the 
general similarity between these approaches to weighting, the results shown below should be 
applicable with little nxxiification to a wide variety of survey projects of both longitudinal and 
cross-sectional design. 

Methodology 

In undertaking this study, I use data collected as part of the Cooperative Institutional 
Research Program (CIRP), a continuing program of research that is sponsored by the American 
Council on Education and the Higher Education Research Institute (HERI) at the University of 
California, Los Angeles* The CIRP freshman survey program annually collects a broad array of 
student background information using die Student Information Form (SIF; see Astin, Panos, & 
Creager, 1966), and is designed to longitudinally assess the impact of coUege on suidents. The 
data for this study arc primarily drawn from tiie 1987 SIF administered to incoming students and 
the 1991 Follow-up Survey of 1987 Freshmen. In addition to these data, further information about 
each student was requested through an institution's CIRP representative, and included information 
on degree earned, number of years competed at the freshman institution, and whetiier or not the 
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student was still enrolled The registrar's survey had a response rate of 68 percent* and these data 
arc an important supplement for the purposes of creating nonresponse weights since this 
information is provided on an unbiased subset of the original follow-up sample (HERI, 1992). 

Sample 

The Student Information Form (SIF) was distributed to campuses in the Spring and 
Summer of 1987 for distribution to college £res^ <men during orientation programs and in the first 
few weeks of fall classes. As pan of the 1987 treshman survey, the 289,875 students at 562 
participating colleges and universities completed the SEP. To reduce the possibility of bias due to 
errors in survey coverage, survey respondents at 172 institutions were excluded from the SIP 
normative population because of a low rate of return from their college as a whole (usually below 
75%). This left 209,627 students at 390 institutions in the national normative population (Dey, 
Astin, &Kom, 1991). 

The Follow-up Survey (FUS), when linked with freshman SIF data, is designed to assess 
a widenrange of student experiences and undergraduate achievements and to provide a longitudinal 
database for studying how different college environments influence student development. A sample 
of SIF respondents in the normative population was drawn using a stratified, random sampling 
procedure designed to ensure an adequate representation of student respondents from different 
types of higher education institutions (HERI, 1992). The stratification scheme classified 
institutions by type and selectivity into one of 23 cells, a sample of 27,1 1 1 students was drawn 
from institutions in the CIRP national norms (i.e., those institutions whose response rates to the 
freshman survey were judged representative of their entering freshman class). This sample size 
was selected based upon earlier Follow-up Survey response rates and was designed to yield a 
minimum of 175 respondents in each stratification cell. 

The Follow-up Survey instrument was sent to students in June, 1991. A second wave of 
follow-up surveys was mailed to non-respondents in mid-August, 1991. The response rate to the 
FUS averaged 20.7 percent. 
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Analysis 

In order to examine the influence of ncMiresponse on results, and the effectiveness of 
weighting procedures to eliminate this influence, the data for specific analyses are drawn only from 
the freshman survey* It is important to remember that the freshman survey data contain full 
infonnation cm all students to whom a survey was sent regardless of whether or not they later 
responded to the follow-up survey* As such, my analysis be limited to those variables which 
appear on either the freshman or registrars* survey. By partitioning these data into respondents (21 
percent) and nonrespondents (79 percent) we can examine how distributions and relationships 
change due to nonresponse. Moreover, by having full data from the freshman survey, we can see 
how well the correct values for the entire sample are reproduced by weighting the respondent data. 

In order to examine the efficacy of the Astin and Molm procedure I conducted several 
analyses which are representative of the kinds of analyses one might see in studies of college 
impact. Specifically I present information on univariate and joint distributions, as well as two 
multiple regressions (one of which is longitudinal in nature, and relies on the subset of data from 
the registrar's survey for the outcome variable). This mixture of analyses is important since 
nonresponse may strongly bias certain statistics associated with a particular set of variables, while 
other statistics based on the same variables may be relatively unaffected. 

Results 

Before addressing the issue of the effectiveness of Astin and Mohn*s technique for 
correcting for nonresponse bias, it is important to first understand the extent to which different 
characteristics predict the likelihood of a student returning a follow-up survey* This is important 
information for it is these characteristics which reveal the ways in which student follow-up samples 
tend to be biased- Table 2 shows the 10 strongest positive and 10 strongest negative correlations 
between student characteristics (measured in 1987) and response status on the 1991 follow-up 
survey (1 = yes, 0 = no). These results show that the two strongest correlates of survey response 
are high school grades and academic self-rating, which suggests that students who tend to do well 
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in educational settings are also those who tend to return surveys (HERI, 1992). Other variables in 
Table 2 which reinforce this interpretation are years studying foreign language in high school, 
scholarly orientation (Astin, 1993), expectation of earning a bachelor's degree, self-rating and 
years of high school study in math, and intending to improve study skills during college. This is a 
critical relationship to note since student retention is an important area of research, and this 
suggests that follow-up surveys will be biased towards those likely to have success in college. 

Student race is another characteristic which predicts survey response. African American 
students are much less Ukely than average to return a survey, while white students are more likely 
than average. In direct percentages, this translates to a response rate among African Americans of 
8.8 percent versus 23.1 among white students. Although it might be tempting to attribute the low 
rate of response among African American students to a hostile educational climate within 
predominantiy white campuses, the fact that the response rate among African American students 
was 1 1.2 percent in white colleges v-.rsus 7.6 in historically black colleges suggests that this 
problem is very complex. The disparity in response rates among students of different races is 
troubling, and an area in need of future study. 

A final pattern in Table 2 which is worthy of note relates to six of the top ten negative 
correlates with response. Taken together, these items show that students who are entrepreneurial 
and oriented toward economic success are less likely to return follow-up surveys than are other 
students. Although this pattern may be related to differential patterns of academic ability (these 
students, as a group, are less likely to be oriented toward academic success), it stands as a 
reminder that there are many possible - and somewhat subtie and unexpected - patterns of 
nonresponse which can lead to biased samples. 

Table 3 shows comparative information on respondents (column 2) and nonrespondents 
(column 3) to the follow-up survey, as well as the complete original sample (column 1) and the 
weighted results based upon respondents only (column 4). Panel A in Table 3 shows the univariate 
disa-ibutions of two variables — high school grades and degree aspirations — while Panel B shows 
the correlation between these two variables. The data on the distribution of high school grades is 

u 
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very different for the respondents and nonrespondents. Whereas 24 percent of all freshmen 
reported a high school grades of A- or better, the corresponding figure for the follow-up 
respondents is 50 percent higher (36 percent). The equivalent proportion among nonrespondents is 
closer to the proportion for the entire population (21 versus 24 percent) which mathematically has 
to be the case since 4 out of 5 students in the freshman population arc follow-up nonrespondents. 
The bias associated with the aspirations for various degrees is somewhat less pronounced yet still 
evident. A comparison of columns 1 and 4 for both variables shows that the weighting procedure 
suggested by Astin and Molm is effective in reducing the bias in the distribution of responses for 
each these variables relative to that which is found in the complete sample. 

Turning now to Panel B we see the effect on nonresponse bias on joint distributions, 
measured here by Pearson's r. By correlating the two variables considered in Panel A we can 
examine the extent to which variables which are individually biased influence their joint 
relationship. In the original sample, the correlation between high school grades and degree 
aspirations is modest (r = .2507). The same is true of the correlations shown in columns 2 through 
4 to the extent that these correlations are essentially equivalent (and certainly statistically equivalent) 
^ the original, correct value shown in column 1. This suggests that even when individual variables 
arc noticeably biased, their relationships with one another tend not to be strongly influenced. 
Earlier methodological studies have come to similar conclusions (Astin & Panos, 1969). 

In order to consider the effectiveness of the Astin and Molm procedure in a multivariate 
context, I conducted two multiple regression analyses which are shown in Table 4. The first 
analyses predicts a student's degree aspiration from high school grades and demographic variables. 
The overall predictability of the dependent variables was similar across the three samples (around 
1 1 percent of the variance in degree aspirations was explained by the set of independent variables), 
and while there are a few differences across the regressions all variables are significant predictors 
in each of the three samples. In comparing the unweighted and weighted results to those from the 
original sample it is hard to identify which comes closest to replicating the original, correct results. 
For example, if we compare across samples using the metric results we see that the regression 
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coefkicients in the unweighted sample tend to be slightly closer to the original values, while the 
constant is closer to the original in the weighted sample. 

Turning now to the longitudinal analysis shown in Table 4, there is less consistency across 
samples in terms of predicting the number of full years of college completed as reported by college 
registrars. Given that this variable is one which is likely to be heavily biased by the patterns 
described in Table 2 this is perhaps not too surprising. Both of the respondent samples 
(unweighted and weighted) suggest that degree aspirations are unrelated to the nunrber of years of 
college completed even though this is significant in the original sample. A similar pattern is found 
for the student race variable. The weighted results fail to show that parental income is a significant 
predictor, while the unweighted results incorrectly identify gender as a significant negative 
predictor. As with the regression predicting degree aspirations it is unclear whether one regression 
- unweighted or weighted - is to be preferred over the other (although the weighted results 
produce a Multiple R value which is slightly closer to that produced in the original, coirect 
regression). Several college impact researchers using other CIRP dataset have not shown a 
tendency to prefer unweighted or weighted analyses, and have characterized the substantive 
differences between these two approaches as trivial (e.g., Pascarella, Ethington, & Smart, 1988; 
Pascarella, Brier, Smart, & Herzog, 1987). 

Discussion and Implications 

These results show that the Astin and Mohn procedure for adjusting for patterns of 
nonresponse is very effective in reducing biases in univariate distributions. The effectiveness of 
this weighting procedure is less clear in adjustiing the results of correlation and regression analyses, 
perhaps in part because these statistics appear to be less systematically affected by bias introduced 
by nonresponse patterns. Moreover, the analyses did not reveal a situation where the weighted 
results were clearly less preferable than the unweighted one. This suggests that the application of 
carefully designed weights is a generally effective method for reducing nonresponse bias. 
Although we cannot be certain that this pattern of results will hold for all analyses under all 
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conditions, previous research seems to suggest that this is indeed the case (Pascarella, Ethington, 
& Smart, 1988; Pascarella, Brier, Smart, & Herzog, 1987; Astin & Panos, 1969). 

The first and most important approach in working with low survey response rates is 
avoiding low survey response rates. The literature on survey research is abundant with good 
infomnation and suggestions on the topic of improving response rates to sur/eys (Dillman, 1991; 
Cote, Grinnell, & Tompkins, 1986; Smith & Bers, 1987). While this literature should become a 
working part of the library of any institutional researcher doing survey research, we need to 
recognize that achieving very high response rates will probably continue to remain problematic 
(especially given the amount of resources available to most institutional researchers to conduct 
surveys of students and faculty). Given these constraints, adjustments of this sort appear to be a 
fairly effective methods of dealing with survey nonresponse. 

Although the Astin and Molm technique is designed to be used in longitudinal research, 
related methods of weighting can be implemented for cross-sectional surveys and are routinely 
employed for this and related purposes (Groves, 1989; Madow, Olkin, & Rubin, 1983; Rossi, 
Wright, & Anderson, 1983). This can be created by defining a number of stratification cells based 
upon characteristics which are 1) known or suspected to be related to nonresponse bias, and 2) 
which are available for the population firom which the original (mail-out) sample was drawn. Once 
this is done, developing weights to adjust the retumed-survey sample to the original sample is 
straightforward (by taking the reciprocal of the ratio of respondents to original sample within the 
stratification cell). Although this sort of post-stratification adjustment may be somewhat less 
precise than that which can be obtained through more complex weighting schemes, they 
nonetheless should be useful in reducing obvious sources of nonresponse bias in cross-sectional 
surveys. It should, however, be noted that all weighting adjustments have subtie and complex 
implications for the analytic (as opposed to more descriptive) uses of survey data (Skinner, Holt, 
& Smith, 1989; Rubin, 1987). Although I have not addressed tiiese issues here, it is important to 
recognize that such issues will become increasingly important as the quality of survey research 
improves within higher education settings. 
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Table 1 

Trendsjnresponse rates to national ACE/CIRP student surveys: 1961 - 1991 



Student cohort Response rate Source 



1961 - 


1962 


58 


Astin, 1968 


1961 - 


1965 


60 


Astin & Panos, 1969 


1966- 


1974 (multiple) 


65 to 40 


Astin, 1977 


1971 - 


1980 


40 


Astin, 1982 


1983- 


1987 


26 


HERI, 1989 


1985- 


1989 


23 


HERI, 1991 


1987- 


1991 


21 


HERI, 1992 
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Table 2 

Top correlates of nonresponse in the 1991 HERI follow-up survey 



Postive correlates 




Neprative corn*latPS 


r 

I 


High school grades 


.180 


Race: African American 


-All 


Self-rating: Academic ability 


.131 


Life goal: Be successful in 








business of my own 


-.091 


Race: White 


.115 








View: There should be laws prohibiting 




Years of HS study: Foreign language 


.107 


homosexual relationships 


-.084 


Orientation: Scholar 


004 




- 08? 


Parent's status: Married, living together 


HQ? 






5?ex' Pemnle 


.091 


Reason for collefie* Imnrove studv skills 


-.078 


Expectation: Will earn BA degree 


.084 


Orientation: Status striver 


-.077 


Self-rating: Mathematical ability 


.082 


Life goal: Be an expert on finance and 






.nmerce 


-.072 


Years of high school study: Math 


.079 








Reason for college: To be able 








to make more money 


-.072 



View: The chief benefit of college 

is to increase earning power -.07 1 
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Table 3 

Effect of nonresponse on univariate and joint distributions 



Complete FUS Follow-up survey 
sample nonrespondents Unweighted Weighted 

A. Univariate distributions 

High school grades 

AorA+ 12.1% 10.2% 19.0% 12.0% 

A- 12.3 11.2 16.8 12.2 

B+ 20.2 19.1 23.6 21.1 

B 20.9 20.9 20.1 23.1 

B- 16.0 17.1 11.6 15.8 

C+ 9.5 10.7 5.1 8.5 

Cor less 8.9 10.4 3.8 7.2 

Degree aspirations 

Post-master's 26.4 25.7 29.2 26.9 

Master's 37.8 37.4 39.5 38.6 

Bachelor's 30.0 30.5 28.2 30.4 

Associate's 5.7 6.4 3.2 4.0 



B. Joint distributions 



High school grades 

& degree aspirations .2507 .2468 .2377 .2529 
Proportion of sample 1^00 .79 ^21 .21 
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Table 4 

Comparison of two sets of unweighicd and weighted regressions 



Complete Unweighted Wei^tcd 
Dependent variable sample results results 



Degree asprioHons (1987) 
Standardized regression coefficients 





25 


23 


25 


Parental income 


.07 


.07 


.11 


Parental education 


.18 


.16 


.12 


Student race: White 


-.16 


-.13 


-.10 


Student sex: Female 


-.02 


-.05 


-.08 


letric regression coefficients 








High school grades 


.12 


.12 


.13 


Parental income 


.02 


.02 


.03 


Parental education 


.09 


.08 


.06 


Student race: White 


-.34 


-.33 


-.23 


Student sex: Female 


-.03 


-.08 


-.13 


Intercept 


4.28 


4.45 


4.28 


Multiple R 


.35 


.33 


.33 




.12 


.11 


.11 


Adjusted 


.12 


.11 


.11 



Number of fail years completed (Registrar's survey) 
Standardized regression coefficients 



Degree aspirations (1987) 


.05 


.02 o 


.00 


o 


High school grades 


.29 


.24 


.28 




Parental income 


.08 


.05 


.03 


o 


Parental education 


.12 


.08 


.08 




Student race: White 


-.04 


-.03 o 


-.03 


o 


Student sex: Female 


.00 c 


-.05 


-.03 


o 


/letric regression coefficients 










Degree aspirations (1987) 


.08 


.03 o 


.00 


o 


High school grades 


.23 


.18 


.23 




Parental income 


.03 


.02 


.01 


o 


Parental education 


.10 


.05 


.07 




Student race: White 


-.17 


-.13 o 


-.12 


o 


Student sex: Female 


.00 o 


-.11 


-.08 


o 


Intercept 


1.70 


3.08 


2.47 




Multiple R 


.36 


.28 


.30 




R2 


.13 


.08 


.09 




Adjusted 


.13 


.07 


.09 





Note: All regression coefficients significant at p < .01 except for those noted with ^. 



